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ABSTRACT 

Computer simulations of three individualized adaptive 
instructional models (AIM) were undertaken to determine if these 
models function as prescribed in Air Force technical training 
programs. In addition, the project sought to develop a user's guide 
for effective understanding of adaptive models during field 
implementation. Successful simulations o£ the Drill and Practice 
Model, the Pacing ;:odel and the Monitoring Model were accomplished 
and operational feasibility vas demonstrated. In addition, a User's 
Guide was prepared and a single integrated technical package 
developed to describe how the models operate. It was recommended that 
simulations of at least two additional models — Concept Acquisition 
and Dynamic Programing~be made and that empirical validation cf 
model effectiveness be conducted employing actual students in real 
time. Such research would allow the comparison cf alternative 
instructional strategies, assess student attitudes toward 
computerized instruction, and provide a means of gaining experience 
in the optimization of instructional strategies prior to widespread 
Air Force implementation. (Author/PB) 
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SUMMARY 



Plobiem 

Providing cost-effective training by ina\imizing student proficiency, while simultaneously minimizing 
student completion time and training costs is predicated upon effective instructional strategies. Analyses of 
past instructional strategy research indicated a current need for refining and developing quantitative 
instructional strategy models capable of adapting instruction to widespread differences among students. 
The question was, "Which adaptive instructional models are suitable for cost-effective technical training?" 

Subsequent to careful analyses and reviews of numerious adaptive instructional strategy molels, 
Phase I research concluded with a recommendation to implement five Adaptive Instructional Models (AIM) 
within the Air Force Human Resources Laboratory's Advanced Instructional System (AIS). Adaptive models 
recommended for AIS field implementation included: (I) Drill and Practice, (2) Concept Acquisition, 
(3) Complex Tutorial, (4) Algorithmic Regression, and (5) Dynamic Programming. Two additional models 
were analyzed and recommended for further research prior to field implementation within tlie AIS. These 
models were ( 1) Natural Language Processing and (2) Automaton Models. 

To minimize adaptive model operational problems, prior to field implementation within the 
Advanced Instructional System (AIS), it was deemed important to test the operational feasibility of the 
selected adaptive models. One method was to computer simulate adaptive model operations to determine if 
they function as prescribed. Hence, three adaptive instructional models were selected for computer 
simulations. 

Objectives of the combined Phase U-III research tasks were to:(l) provide computer simulations of 
three previously developed adaptive instructional models, (2) demonstrate adaptive model operational 
feasibflity for use in the Advanced Instructional System (AIS) by simulating each adaptive model with Air 
Force personnel data, and (3) develop a user's guide for effective understanding of adaptive model 
operations for successful field implementation. 

Approach 

Adaptive models selected for computer simulation were: (1) Drill and Practice, (2) Pacing, and 
(3) Monitoring (Algorithmic Regression). Detailed procedures, assumptions and parameter values employed 
in each of the three computer simulations are presented within the text. To facflitate comprehension, a 
user's guide approach to include detailed examples of reports to students, instructors and instructional 
managers has been utflized as a presentation format. 

Results 

Successful computer simulations of the: (1) Drill and Practice Model, (2) Pacing Model, and 
(3) Monitoring Model were accomplished. Model operations, recommendations for investigating parameter 
values odier than those employed in the computer simulations and diffei«Mitial methods for model 
validation were included in the User's Guide. Additionally, the Phase II model simulations were combined 
with the niase III Guide to provide a single integrated technical package for each model simulated. The 
technical package consists of: (1) a brief description of the model simulated, (2) a simplified explanation 
of how the model operates, (3) provides a scenario, using the Air Force data actually used in the simulation 
so the reader has a concrete, step by step example of how the model uses information to prescribe and 
adapt instruction to individual student performance, (4) gives examples of instructional objectives and 
instructional items used to attain such objectives, and (5) outputs reports to students and instructional 
personnel. 

Conclusions and Recommendations 

Computer simulations fmdings demonstrated the operational feasibility of three adaptive 
instructional models. It would be prudent of the Air Force to computer simulate at a minimum two 
additional models: (1) Concept Acquisition and (2) Dynamic Programming Models. Concept acquisition 
models formulate individualized strategies to ensure concept understanding in the least time possible. 
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Dynamic Programming models seek to optimize both student progress in real time as well as minimize total 
instructional costs for the entire Instructional system. 

It is furtlier recommended that empirical v.Oidalion of model effectiveness be conducted employing 
actual students in real time. Benefits of such research would include comparison of alternative instructional 
strategics, student altitudes toward computerized instruction and experience in optimizing instructional 
strategy effectiveness before widespread Air Force application. 



PREFACE 



The present study is one of a series of reports presenting adaptive instructional 
model research and computer simulations of three Adaptive Instructional Models (AIM). 

To facilitate the widespread utility of these computer simulation findings, 
contained herein is a step-by-step user's guide. Further defmitive fmdings are reported in 
AFHRL-TR-72-50 (I). Analysis and Development of Adaptive Instructional Models for 
Individualized Technical Training; Phase 1.'' Research was accomplished in support of the 
Advanced Instructional System under Job Order 1193/OB/05. The research was 
conducted by the Center for Computer Assisted Instruction, Florida State University, 
Tallahassee, Florida, under contract F33615-71-C-1277 with the Air Force Human 
Resources Laboratory, Technical Training Division, Lowry Air Force Base, Colorado. Dr. 
Duncan Hansen was the Principal Investigator for Florida State University. Mr. Joseph 
Yasutake was the Division Project Scientist and Dr. Gerard M. Deignan was the Division 
Task Scientist. 

Although many individuals contributed significantly to the successful completion 
of the Phase II and Phase III research efforts. Major Roger Grossel, Captain Edward 
Gardner, and Mr. Joseph Lamos are recognized for their constructive criticisms and 
suggestions. Dr. Robert Tennyson was indispensable in guiding this work to fruition. 

The summary and other portions of the Phase I, II and III reports were prepared by 
Dr. Gerard M. Deignan. 
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A GUIDE TO COMPUTER SIMULATIONS OF THREE ADAPTIVE INSTRUCTIONAL 
MODELS FOR THE ADVANCED INSTRUCTIONAL SYSTEM PHASES II AND III 



I. INTRODUCTION 

Euucatlonai institutions are faced with the challenge of providing an adequate education to every 
student At the same time, these institutions must design instructional systems that are cost effective. 
Research during the cast decade in educational environments has demonstrated that indhidualized 
instruction is the most effective system in optimizing the learning proceu. The implementation costs of 
such a system have been prohibitive except in small experimental situations. Recent advancements in 
technology, especially in computermanaged instructional systems, show that the goal of individualization 
can be obtained within acceptable cost structures. 

The Air Force training system has taken the initiative in designing an educatiorial institution that 
meets the societal demands of quality learning for each individual while maintaining acceptable costs. The 
Air Force Human Resources Laboratory is currently developing an Advanced Instructional System (AIS) to 
be implemented in the technical training environment at Lowry Air Force Base (APB). The AIS project 
began with a series of "^esearch studies to detemiine: (I) the basic speciflcations of the system, (2) the 
design of instructional models that would adapt to individual perfomianc8,(3) the methods for evduating 
and measuring students given a totally individualized system, and (4) optimal instructional media. Tht« 
inve:tigations are to be used in the development of AIS courses at Lowry AFB. 

The specifications study outlined a format for the development of seven AIS subsystem components. 
The present study reports the final design phase of one subsystem component - Adaptive Instructional 
Models (AIM). 

Adaptive Instmctional Models constitute the means by which instructional tasks, materials, media 
and resources are continually tailored or adapted to match the changing instructional needs, skills and 
interest motivations of individual students. To obtain a deeper understanding of the functions and 
interrelationships among various adaptive models, the reader is referred to a previous technical report 
(AFHRLTR-72-50 (I)) entitled, "The Analysis and Development of Adaptive Instructional Models for 
Individualized Technical Training - Phase I. (Ha isen, Brown, Merrill, Tennyson,Thomas, A Kribs, 1973). 

Seven adaptive models were designed and categorized into three general areas: (1) bask skBI 
stmtegieS'ixm and practice, and pacing models; (2) btstmctional strategies-conctpU nile using, and 
problenvsoWing models; and (3) management smr/ei^es-monitoring, automata and dynamic programming 
models. These models were presented in the Phase I report, which discussed in detail the rationale, design 
and structure of each adaptive model. 

Phase II, of the AIM contract, was to demonstrate the feasibility of using the models in the AIS. This 
report (AFHRL>TR-72- SO (II)) is a narrative description of the assumptions and rationale uaed in designing 
the three simulated models. This document presents the simulati'7ns,with emphasis on the characteristics of 
the models and samples of reports generated from the simulations. Concluding the report is a section on 
what was learned from th^e simulations and recommendations for extended study. 

Overview of AIS 

The AIS is a prototype, computer-based, individualized training system ai.d research facility, designed 
to improve and maintain the cost-fffectWeness of technical training. Individualization of technical 
instruction, computer management of instructional resources, and evaluation of instructional strategies are 
some of the AIS provisions expected to contribute considerably to the effectiveness and efflciency of 
technical training. Additionally, the AIS will serve as an evaluative test base for promising future 
instructional innovations. 

Initially, the Advanced Instructional System will be implemented in three Air Force technical 
courses: (a) Precision Measuring Equipment, (b) Weapons Mechanic, and (c) Inventory Management. The 
AIS is designed to operate as a totally integrated compute r-baaed system, capable of training approximately 
2,100 students in the three courses with a projected 25 percent average reduction in training time. Time 
savings will be accompanied by training proficiency equal to or better than that of non-AIS graduates. 
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Ovmkwof AIM 

Adaptive InitnicUonal Models ire the metni for inalyzlng student chtncteriitki within the context 
of ipectflc training taiki and objecttvet for the putpoie of preacribini dllTerendal inttnictional conditions 
and leiources for maximum training efnciency. These Instructional models provide adaptability becauK 
instnictional aatignmenti and conditions are based upon and mediated by the characterls^cs and 
performances of each student. Moat Importantly, the models provide for a cybernetic, or feedbaclci proceu 
by whkh student data, the effectlvenott of instructional treatments and resources are constantly updated to 
improve the overall uaining system. 

AIM obfecihet The objectives of the AIM project contribute directly to the goals of the Advanced 
Instructional System, namely* individualization of the training process, computer management of resources, 
use of ost'effectlve multi-media approaches, and uaining systems modular!^. 

Specifically, the objectives of AIM are stated as foltnw^: 

1. To provide monitoring of the student*s rharacteristics and performance. 

2. To provide a set of training decision rules that optimize student motivation and proflciency. 

3. To provide a decision allocation procedure that optimally assigns instructional media, material, 
and incentive rewards according to the student's characteristics and performance. 

4. To provide predictions of perfoimanoe and time parameten for the student. 

5. To provide for the sdiedulmg of all instructional resources to lead to a minimization of training 

costs. 

6. To pro/ide procedures for monitoring and evaluating alternative adaptive model effectiveness 
according to appropriate matches among training task requirements, instructional resource availabflity, and 
specific individud differences. 

Puipoee of AIM Shnulationa 

Simulation is a technique for modeling a particular process while manipulating variables so that 
possible outcomes can be observed. It is possible to represent and evaluate a reid world situation without 
actually modifying or developing the environment being studied. The simulations described in this 
document allowed for the detailed specification and opera tionalism of AIMs in a compu^r-based, 
individualized, multimedia environment. This saved costly trial-and-error type efforts of development by 
allowing planning to proceed the development effort. 

All of the necessary input and ouput information elements of each model's functiorud environment 
were demonstrated by the simulations. Operationally defined statements of model information 
requirements provided a link between the models and other AIS components, including the computer and 
computer software subsystems as well as the instructional materials, and media subsystems. Input data 
elements included: (a) student preassessment data; e^.. Airman Qualification Exam (AQE) scores, 
(b) student performance data; , perfomiance within lessons and instructional blocks; and (c) incentive 
histoiy; e^, motivational preferences. An important coniiiuutlon of the input and output data 
specifications in the AIM computer simulations was the explicit representation of the adaptive process. The 
representation of the ad4)tive process by which students' data were appropriately used as parameters in the 
context of AIS decision rules and procedures was an implicit goal in this effort. Lastly, the three AIM 
computer simulations provided a direct indication of how AIMs can be nested; ie, how one model can be 
controlled by another model. Thus, the instructional application of one model may be initiated by a 
higher-level model. For example, the sinulated monitoring model may serve as a macro-decision maldng 
model for sdecting a particular instructional treatment, such as the drill and practice model, which in turn 
may utilize the performance-contingent pacing model. Nested functional relationships are important 
features in the development of a range of adaptive models that can be coordinated within a master model to 
provide adaptive instructional monitoring, control, and decisions in a computer-baaed instructional system. 

AIM SlmuIatioQ Methodology 

Considerable methodology for computer simulations exists, but computerized simulations have 
seldom been applied in the training area, llie methodology used within this project represented concepts 
derived from computer simulation techniques, task analysis, and instructional systems theory. The sequence 



developed in this project followed seven steps: (a) specification of data elements and reports, 
(b) nowcharting of major components, (c) specification of typical protocols and input-output 
relationships* (d) speciHcation of the quantitative processes in the model, (e) program implementation, 
(f) evaluation and revision, and (g) documentation. An explanation of ecich one of these steps follows: 

Step 1 : Spedflcation of Data Elements and Reports 

The purpose of this step was to specify what data elements and records were required. The data 
elements were; 

1. Pteassessment data. The preassessment data on a given student were categorized by aptitude, 
personality, and performance measures. 

2. MacrO'perforrmnce data. The system provided data on the lesson and block criterion tests 
according to task type and media as opposed to individual test performance indices. 

3. Micro-performance data. On speciflc lessons the system provided response records which include 
item error data latencies, confidence, and state anxiety measures. 

4. Reports, Reports were generated at three levels: (a) student level, which provided a prescription 
and feedback on performance in the lesson, (b) instructor level, which provided a report on students 
participating in a class, and (c) management level, a report on students participating in a block of 
instruction. 

In addition, there were, when appropriate, specific item-by-item protocol report forms generated by 
the computer simulation. As will become clear in subsequent sections, each of these elements varied from 
model to model. 

Step 2: Flowcharting of Major Components 

For each model, the major components were identified, interrelated, and given a quasi-quantitative 
description. In order to maintain consistency, the following components were considered in each of the 
simulations: 

1. Initializing components. Provides procedures for initializing or updating student data elements 
prior to instruction. 

2. Lesson prescription components. The process by which a lesson was appropriately composed for a 
student. 

3. Lesson process components. Permits adaptive interactive processes between students and the 
computer under the control of the model. Interactive processes refer to the frequency, format, and content 
of messages flowing between students and the computer. 

4. Lesson evaluation components. Provides evaluative feedback to the student, instructor, and 
adaptive model. 

5. Special logic and calculation components. Lessons requiring specialized logic or calculation 
components are separated and clearly identified. 

Step 3: Specifk^ion of Typical Piotocok and Input/Output Relationships 

Given the availabflity of the preceding components, it was necessary to specify and implement the 
flow of infomiation as it proceeded through the input, processing, and output states of the simulations. In 
Step 3 the exact protocol that represented the learning process for a given student was specified in detail. 
The specification process can be stated as.fdlows: 

1. Specify the critical data elements for the protocols. The subset of data elements essential for 
protocols at given states in the model were identifled. 

2. List protocols at each major component point in the lesson format. The accumulation of data in a 
particular protocol was specified. 

3. Specify the input, output devices which are controlled or monitored by the adaptive model 
Providing a statement of context for the simulation was a main goal of the simulations. 
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Step 4: Specification of the Quantitative Processes in the Model 

The next step consisted of identifying, from a quantitative point of view, the constraints, constants, 
and parameters that would tit within appropriate mathematical, deterministic, and stochastic functions 
required for a realistic computer simulation. Realism was furthered by using appropriate Air Force 
information such as student aptitudes, course and lesson characteristics, concept topics, and objectives. In 
addition, data obtained by Vitoia and Alley (1968) were used to provide Air Force trainee preassessment 
data for the simulations. The following specifications were given: 

1. Specify quantitative processes. Each major component within the three simulations was 
quantitatively identified. 

2. Specify individual characteristics. Individual differences likely to be encountered in an Air Force 
training environment were represented. 

3. Specify random distributions. Distributions to be used for any of the random processes, in terms 
of distribution statistics which include means, variances, and ranges, were estimated. 

StepS: Program Implementation 

At this point, a professional programmer implemented each of the simulations in A Programming 
Language (APL), a programming language which allows for student interaction simulation and simplifies 
complex mathematical functions. In some cases, revisions of the programs were based on the computer core 
requirements. These revisions were made with the intent of allowing for a minimum amount of core 
requirements so as not to overtax the ultimate AIS computing system with additional computing 
equipment demands. 

Step 6: Evaluation and Revision 

For each of the simulations, evaluation was pursued in terms of checks on student, instructor, and 
management reports. These reports were compared for realism as to score values and actual distribution; 
these enor checks provided the input for reprogramming. In the vast majority of cases, the reprogramming 
consisted only of a change in the parameters for a given distribution or reformatting of the report labels. 

Step 7: Documentation 

In order to complete the simulations, each APL program was documented in detail. The remaining 
chapters of this report detaO the three simulations. Each simulation implicitly assumed the AIS 
environment in which mastery by students of well-specified objectives was required, and students could not 
proceed until mastery was attained. 

AIM Simulation Assumptions 

The three simulated models, drill and practice, pacing, and monitoring, were developed according to 
rationales based upon empirical data and assumptions derived from instructional theories. The purpose of 
this section is to provide an overview of the simulation variables to show the similarities and differences of 
the three models. 

Drill and practice simulation. The drill and practice model was designed to improve a student's 
accuracy in performing various skills. An individualized prescription of practice items was determined for 
each student in accordance with three basic assumptions: 

1. Not all students require practice on the same set of skills. Consequently, items were included in 
the prescription in proportion to ihe student's prior mastery of the related skill as determined by a pretest. 

2. Not all students learn at the same rate. Therefore, some students required more practice to reach 
mastery than other students. An index of student characteristics, based on preassessment data, was used to 
predict variadons in the amount of practice required and prescribed items. 

3. Some tasks are more difficuU than other tasks and, consequently, require more practice to reach 
mastery. 

4. An index of task characteristics which was predetermined on the basis of empirical evidence was 
also used in prescribing the number of practice items to be presented. 
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Underlying all drill and practice models is the fundamental assumption that a student's performance 
improves with practice until it approaches perfection. Therefore, the probability that the simulated student 
would respond correctly to an item from a particular task was incremented after each presentation of items 
for that task. Since feedback was provided after each response, even incorrect responses were assumed to be 
learning experiences and, accordingly, the probability of a correct response was increased regardless of the 
accuracy of the previous response, In addition, it was assumed that the amount learned from each 
presentation of an item varies from student to student. To incorporate this assumption into the model 
simulation, the increment in the probability of a correct response was a function of an index of student 
characteristics, namely his AFQT score. The following formula was used to determine the probability (P) of 
a conect response on the n^^ presentation of an item related to a speciflc task: 

Pfi ~ + (1 - Pn-1) (.003 + .OOlZ) 

where Z is the z-score equivalent of the student's Armed Forces Qualiflcation Test (AFQT) score. The 
initial probability of a conect response was set at either 0.50 or the proportion conect on the pretest 
depending upon which is greater. Using the above formula, the probability of a conect response approaches 
but never reaches 1 .00. 

A flnal assumption of the drill-and-practice model was that performance would drop after practice 
ceases. Consequently, the model includes a review component. Items were selected for review on die basis 
of the relative importance of the associated objective (task) and the difficulty the student had with that 
item during the ddll. Priority for selection was given to more important objectives and more difflcult items. 

Pacing simulation The design of the adaptive pacing model was founded on two assumptions. First, it 
was assumed that self-paced, problem-solving rates would not yield an increase in problem-solving speed. 
Secondly, it was assumed that shaping a student's behavior by requiring him to respond at increa^ngly 
faster rates would ultimately result in an increase in problem-solving speed. In terms of the construction of 
the simulation, several further assumptions were made. The probability of a conect first answer was set at 
0.5; ie., on the first attempt at the item the student had a 50 percent chance of answering it correctly. The 
probability of getting an answer wrong decays exponentially with the number of answers. Thus, the 
simulated student began responding with conect answers at a chance level and gradually improved with 
practice. And lastly, the student's latencies were assumed to decay exponentially with the number of 
answers. Since no student data other than that generated by him while performing the task were used by 
the pacing simulation, it was not possible to describe any particular student characteristics for this model. 
However, the pacing model was assumed to operate in conjunction witli the drill and practice model; the 
student characteristics discussed in the preceding section describe the students in the pacing model. 

Monitoring model simulation This simulation ^as designed as a macro-model to be used as a monitor 
of the student's progress through an entire course. The previous simulations, drill-and-practice, and pacing, 
were micro-models concerned only with specific areas of a course. Therefore, the monitoring model used a 
variety of input variables to determine the student's individual assignment based on which micro-model, 
which media mode, and which measuring procedure was involved in the decision of instructional sequence. 
Equations, with assumed beta weights were generated for each of five time and five mastery treatments per 
lesson. Student characteristics were calculated using fifteen measures (f.^.. Armed Services Vocational 
Aptitude Battery (ASVAB) scores, AQE scores, task indices) which were conelated for each student's 
individual measures with ail the other students' measures. The student's conelations were used in the 
equations to determine predicted score and time. The highest score of the five treatment equations was 
selected for the instructional sequence, along with the fastest time equation. Mastery for a given lesson was 
set at 80 percent conect on the criterion measures. The student's original correlation scores stayed the same 
within the simulation, but for each new lesson, the equations were different, requiring further calculations 
for assignment. 

The monitoring model was designed according to these basic assumptions of instructional science: 

1. Alternative instructional treatments can be developed for measured student differences. 

2. No one treatment is **best" for all students, 

3. There exist measurable individual difference variables which can be used to predict the most 
efficient treatment for a given student. 

4. Individualized treatment assignments can reduce the time and cost of training without sacrificing 
Q performance. 
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II. DRILL AND-PRACTICE MODEL 



Drill-and-practice is defined as an instructional process which presents to the student a series of 
problems that require responses according to specified criteria. Two components of this model are 
immediate analysis of the student's response, and feedback to provide knowledge of results, llie computer 
model can adapt the type and amount of problem practice on the basis of student characteristics, prior 
performance, and other training variables such as incentives. The goals of drill-and-practice computer 
strategies are: (a) to improve the student's accuracy, and (b) to increase his performance speed. 

The first portion of this section presents a description of the data elements (the input and output 
measures) used in the simulation of the drill-and-practice model. A second section of the chapter is devoted 
to the major components of the simulation structure and the five types of output reports which were 
simulated. Tables of simulated reports, with annotated items, are presented for the Student Protocol 
Report, the Report to the Student, the Report to the Instructor, the Management Report, and the Item 
Summary. The chapter concludes with suggestions for future validation of the model. 

The drill-and-practice model can be used for topic areas within each of the three courses (Inventory 
Management, Precision Measuring Equipment, and Weapons Mechanic) identified for development in AIS. 
The use of preassessment data, pretest scores, and task characteristics to generate lesson prescriptions 
individualizes the instruction to match both the student's and the institution's goals and objectives. This 
model also involves several options with respect to the sequencing of topics (objectives) and the amount of 
practice provided. The amount of practice is determined on the basis of: (a) preassessment data, (b) task 
characteristics, and (c) performance within the drill. 

Data Elements 

Data elements are defined as measurements and variables which are used to assess and prescribe the 
student's behavior. For the simulated drill and practice model three assessment variable groups were 
designed. First, preassessment data were used to identify the student's entering behavior to prescribe the 
initial instructional sequence. Second, task characteristics were the operational variables within the task. 
The third group, performance data, represented the student's actual behavior during the task. 

Preassessment data. Measures of Airman pre-entry instruction behaviors were classified into 
categories: student profile and characteristics data. Contained within the student profile were Air Force 
persorme! data such as the AQE scores, AFQT scores, educational level, prior work experience, performance 
on relevant courses, and biographical information. Student characteristics data included course specific 
aptitude scores, cognitive style indices, affective, attitudinal, and interest measures. These measures were 
accessed by the drill-and-practice model to formulate prescriptions, ultimately determining the amount of 
practice the student received and the size of the active item pool (a reflection of the individual's learning 
processes). 

Task characteristics. A task index was used to determine the number of problems to prescribe for 
each objective. In general, the higher the index, the more problems a student was assigned. This index was 
directly related to the difficulty or importance of the objective. In a review session after drill and practice, 
an index of the relative importance of each objective was also used to select problems for the review. In 
addition, an index of the student's performance on each item within the drill was utilized to set priorities in 
the selection of problems within the objectives. Thus, the more important objectives were given priority 
over the less important objectives, and a problem which a student found more difficult was given priority 
over an easier item. 

Performance data. Several indices of student performance within the drill were collected and used in 
the simulation. The student's history of correct responses to each problem was considered the most 
important variable. The number of consecutively correct responses was used to determine when the student 
had mastered a given problem; for this simulation the criterion was arbitrarily set at two consecutive correct 
responses. In addition, the percent correct of all responses to a given problem was computed to determine if 
the problem should be included in the final review. The number of times a student was presented a problem 
before he reached mastery was also collected. These data were used to summarize the performance in a 
student protocol report. 
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Another index of student performance which can be collected by the drill-and-practice model is 
response latency (time to perform a given amount of instruction). Mean latencies can be computed for each 
problem, each objective, and for the total drill. Latency data has not been included in the drill-and-practice 
simulation due to the obvious empirical uncertainties. But latencies could become part of the index for 
masteiy , as could the two consecutive conect responses. 

Simulation structure. The simulation of the driU-and-practice model was programmed in APL and run 
on an IBM 1500 instructional system at Florida State University CAI Center The program was repeated 90 
times with each repetition representing the perfonnance of one simulated student between the pretest and 
the posttest Five instructional objectives from Block III of the Precision Measuring Equipment course were 
used in this simulation. 

Miyor Components of the Drill and Practice Simulation 

The prescription, drill process, and review of the simulated model included the use of student 
preassessment and performance data and task characteristics for the construction of individualized lessons 
to be presented within the drill sequence. Decisions were made regarding the number of problems to be 
presented in a lesson and the sequence of problem types according to perfonnance objectives. The 
simulation approach centered on the concept of instructional objectives and instructional prescriptions. The 
five objectives were representative of those found in the PME course, ^^ich range from 5 to 30 for a block 
of instruction. Each objective had a set of problems designed both to instruct the student, and to measure 
his masteiy of the objective. The number of problems in each objective's set was arbitrarily fixed for this 
simulation. The number of problems varied across objectives as a demonstration of the model's 
generalizability. 

For the purpose of the driU-and-practice simulation, the following preassessment measures were 
collected: (a) AFQT scores, (b) the AQE Mechanical Aptitude Subtest; (c) the AQE AdmlnisUative 
Aptitude Subtest; (d) the AQE General Aptitude Subtest; (e) the AQE Electronic Aptitude Subtest; 
(0 ASVAB Word Knowledge Test; and (g) the ASVAB Arithmetic Reasoning Test. In the simulation only 
the Airman's AFQT score was used as a variable in the decision process. The means, standard deviations, 
and intercorrelation matrices obtained from a sample of approximately 1,000 basic airmen (Vitola & Alley, 
1968) were used to generate realistic preassessment data for the simulated students. These preassessment 
data were also used: (a) to establish the probability that the simulated student would respond conectly, 
and (b) to estimate the time for the student to complete the prescription. This involved the use of 
regression techniques plus stochastic processes using a uniform distribution. 

Prescription An individualized prescription was generated for each student per objective on the basis 
of his pretest scores, task characteristics, and student characteristics as represented in Figure 1. The index 
of pretest score was defined as 1 minus the proportion corcect of those problems in the pretest relating to 
each objective. For the purposes of the simulation, the pretest scores were randomly generated for each 
objective. It was assumed that even a bright student might never have had any prior experience with a 
particular objective and would perform poorly on the pretest, therefore, the following conditions were 
imposed upon the simulation exercise. For a given simulated student, there was no intentional correlation 
among scores on the five instructional objectives, nor was there a conelation between the total pretest score 
and any preassessment data for that student. 

The index of task characteristics in the model has been called a selection index and was 
predetermined for each objective by the course development component of AIS. This task selection index 
was directly related to the importance of the objective or the difficulty in learning the objective. 
Accordingly, more practice was provided for objectives with higher selection indices. For the purposes of 
the simulation, the task selection indices were randomly generated numbers ranging from 1 to 5. 

The third variable used to generate prescriptions was based on student characteristics and was an 
index of the amount of practice a student needed to attain masteiy. For the purposes of this simulation, 
^ the student characteristic index was a function of his AFQT score and ranged from 1 to 5. (On the baas of 

future results obtained from employing this model with actual students, the student characteristics index 
can be refined to reflect additional preassessment data.) 
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Figure 1. •-First component of the drill- and-practice simulation- 
Prescription. 

To determine the number of practice problems to present to a student for each objective, the 
following formula was used: (1 - pretest score) x (selection task index) x (student index). The result of this 
multiplication was rounded up to the next whole number. 

The model required that for each objective, there must be a large pool of problems from vAiidi to 
select the practice problems. The specific problems prescribed for each objective were randomly selected 
from this pool. If there were fewer problems in the pool than the prescription required, the model would 
select an problems available. 

The next step in the model was to divide the prescription into lessons of approximately 20 to 25 
problems each. Whenever possible, all problems relating to a particular objective were assigned to the same 
lesson. The problems for each lesson composed a lesson pool and were randomly sequenced. 

Drill process. In order to optimize the frequency with v/hich a problem was presented, a subset of the 
lesson pod called the "active pool** was designated (Figure 2). A number specifying the active pool size or 
optimal block size could be determined by statistical regression techniques on the basis of both student and 
task characteristics. For the purposes of this simulation, the active pool size was a randomly generated v 
number from 10 to 1 5, since the empirical relationships were unknown. 

In the simulated drill process, the active pool was filled with problems from the lesson pool. The 
items were presented sequentially, looping back to the first item after the last problem in the active pool. 
When the student met the criterion of correctly responding to two successive presentations of a given item, 
the item was deleted from the active pool and moved to a review pool. A new problem was brought from 
the lesson pool. When there were no more new problems in the lesson pool, the active pool was allowed to 
decrease to a minimum size of 5 after which an item was selected from the review pool to maintain this 
minimum size of the active pool. This process continued until all problems had been mastered. The same 
procedure was employed for the remaining lessons. 




14 




Begin presenting 
Items in "active pool" 



Delete items as 
mastered 




No 



Yes 




No 



Figure 2,--Second component of the drill and practice simulation- 
Drill process. 

For the purposes of the simulation, the "correctness" of the simulated student's response was a 
randomly-generated number based on the probability that the student would respond correctly on that 
particular presentation. At the beginning of the simulation, the probability that the student would respond 
correctly to any given item was the pretest score for each objective, but no less than 0.50. For each 
presentation of a problem for a given objective, this probability was incremented as a function of the 
student's AFQT score. This probability was incremented separately for each objective and asymptoted less 
lhan one. For both correct and incorrect responses, the probability that the student would answer correctly 
on the following presentation was increased, since it was assumed that learning could occur from an 
incorrect response if feedback was provided. 
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Review. When the simulated student had completed all the lessons composing the prescription, a 
review lesson was constructed (Figure 3). The criteria for selecting items for the review lesson were: (a) the 
importance of the objectives to which the problem belonged, and (b) the performance of the student on 
that problem within the drill lessons. Problems belonging to objectives which have higher relative 
importance were given priority. Also, within each objective, the problems with which the student had the 
greatest difficulty (as defined by more than the average number of presentations required to reach mastery) 
were given priority over the easier problems for that objective. The process for presenting the problems in 
the review lesson was similar to the process employed within each of the drills except that the criterion for 
deleting an item from the active review pool was one correct response. This review lesson was composed of 
25 problems which were randomly ordered and presented until the student had answered each one 
correctly. When the student had finished the review lesson, he was ready to go on to the posttest. 



© 



Select items 
for review 



Present re- 
view items 




Figure 3.— Third component of the drill and practice simulation* 
Review. 
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Output Reports 

Five reports were designed for the drill-and-practice model simulation to illustrate the type of 
information which could be collected. The five sample reports from the simulation model wUl be presented 
and discussed. They are: 

1. The Student Protocol Report 

2. The Report to the Student 

3. The Report to the Instructor 

4. The Management Report 

5. Item Summary 

Student protocol report. The student protocol report was a detailed record of a students 
performance as he progressed through the drill. It was made up of four major sections: (a) initialization 
information, (b) detailed response records, (c) item summary, and (d) instructional objective summary. 
Each of these sections will be presented in a simulated table on one student including annotations of the 
tabled information. Table 1 shows a simulated protocol for a given airman. The initialization information 
identified the student and generated his individualized prescription of practice items. This data allowed the 
instructor to review the type of decisions made for each student. 

The second pnrt of the student protocol report consisted of a series of records corresponding to each 
response the student made as he proceeded through the drill. The detailed response records, as is shown in 

Table 2, began with, "The assignment for lesson 1 is "A prescription may have been broken into two 

or more lessons, and this line indicated the number of items from each objective assigned to lesson 1, 2, etc. 

Following this information on student responses, an item summary was printed out as simulated in 
Table 3. 

Finally, there was an overall summary of the student's performance. The simulated objective 
summary is simulated Table 4. 

Tables 1 through 4 illustrate with simulated data, the components of a complete student protocol 
report in the drill-and-practice model. A second set of reports for the drill-and-practice model was that 
report intended for the student himself. 

Report to the student. The student report was divided into two sections; the first section was printed 
out at the student's terminal or on the nearby line printer at the beginning of the drill to inform him of the 
prescription he was to follow. In addition to student identification information, it provided the name of the 
lesson, the number of items assigned, and the estimated time for the student to complete the prescription. 
This preinstructional report to the student is simulated in Table 5. 

The second part of the report to the student was printed after he had completed the prescription to 
inform him of his progress and performance. This again included student identification information. The 
evaluation was intended to give the student an indication of which objectives he needed to emphasize while 
studying for the posttest. A sample is shown in Table 6. 

These preinstructional and postinstructional reports were made directly to the student. A third major 
reporting instrument in addition to the Student Protocol Report (for an individual student) and the Report 
to the Student, is the Report to the Instructor. 

Report to the instructor. The instructor report summarized the performance of 15 students on the 
same drill and practice lesson and was designed to facilitate the instructor's evaluation of their progress. A 
table of pretest results for each student as well as the mean and standard deviation is provided. A similar 
table summarized the student's performance in terms of the percent correct. Table 7 depicts simulated 
information on a report to the instructor. 

Management report. The fourth type of report simulated, the management report, was calculated and 
printed after each group of 45 students had completed the lesson (Table 8). This report summarized their 
pretest results, the number of items assigned, and theii performance within the drill. In addition there was a 
cumulative report providing similar information regarding all prior students who had completed this lesson 
(Table 9). The management report facilitated the evaluation of the instmctor*s performance. A secondary 
purpose of this report was to aid in the evaluation of the course for revisions. 

Cumulative management report. Except for listing the participating students' AFSNs, the information 
contained in the cumulative management report was the same as that in the management report (Table 8). 
The data reported was cumulative on all students who had taken the particular lesson and who were 
included in the current management report. 
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TABLE 1 



Simulated Drill^anchPractice Student Protocol Report 
Part 1: Initialization Information 



AFSN ^ NAME 
768901 Jones, A.E. 

Preassessment Data^ 



COURSE BLOCK LESSON INSTRUCTOR DATE 
PME III 01 Clark, D. 08/24/72 



AQE Mech AQE ADM AQE Gen AQE Elec AFQT Verbal Arith 
61.80 73.26 76.78 55.47 49.56 21.15 18.83 



LESSON TOPIC: Resonance in Tuned LRC Circuits 






7 

Instructional Objective Code 


1 


2 


3 


4 


5 


Objective Selection Index 


1 


2 


2 


2 


3 


9 

% Correct in Pretest 


96 


66 


24 


26 


7 


10 

Number of Items Prescribed 


1 


3 


7 


7 


12 


Number of Items Assigned 


1 


3 


7 


7 


12 


12 

Total Number of Items in Pool 


30 










Nutnber of Items in Active Pool " 


13 











^AFSN - Armed Forces Serial Number. Due to t/stem limitotions, only six digits rather than a full social security 
number were recorded. These numbers were randomly generated for the simulation. 

^Course. Course titles were abbreviated. 

^ Block. Courses were divided into blocks and labeled with Roman numerals. 
^Lesson. The specific lesson being tau^t was recorded by number. 

^Preassessment data. Preassessment measures were generated to match test statistics which were derived from a 
sample of 1,000 airmen ( Vitola & Alley, 1968). 

^Lesson topic. Lesson content was briefly described. 

Instnictional objective code. Each instructional objective was numerically coded. 

^Objective selection index. The selection index numerically assigned each objective according to its difficulty and 
importance. More items were assigned for objectives with high selection indices. For the purposes of the simulation, the 
selection indices were randomly generated numbers from 1 to 5. 

^Correct in pretest. The percent correct of responses to the pretest wasrecorded separately for each objective. For 
the purposes of the simulation, these scores were randomly generated. 

Number of items prescribed The number of items prescribed for each objective was determined by the following 
formula: (1 - pretest score) x (selection index) x (student index). (For the purposes of the simulation, the student index 
was a function of the AFQT score only.) 

Number of items assigned. Generally, this was the same as the number of items prescribed unless more items were 
prescribed than were available. 

^ ^T otal number of items in pool. This was the sum of items assigned across all the objectives. 

Number of items in active pool. The size of the active pool was determined on the basis of both student and task 
characteristics. The active pool size in this simulation was randomly varied from 10 to 1 5. 
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Simulated Drill-and-Practice Student Protocol Report 
Part 2: Detailed Response Records 



THE ASSIGNMENT FOR LESSON 1 IS . . . 



OBJ 

# of items 1 


1 OBJ 


2 OBJ 3 OBJ 
7 . 7 


4 OBJ 5 
12 












DETAILED RESPONSE RECORDS 






TRIAL^ 


ITEM ^ 


OBJ. ^ 
CODE 


# PREV'* PREV^ 
ENCOUNT. COR. 


INTERVEN-*^ 
ING ITEMS 


PROB' 
COR. 


a 

RESP 


1 
2 
3 
4 


3 

29 
6 

41 


5 

3 
4 
5 


. o ooo 
. o ooo 


1 
2 
3 
4 

• 


0.50 
0.50 
0.50 
Q.50 


0 
0 
1 
0 

• 


• 

• 


• 

« 


• 
• 


• • 


• 




• 



PREV' 

msT. 

0 
0 
0 
0 



96 14 5 9 1 5 0.59 



^ Trial. Each presentation was numbered sequentially. 
^Item. The number of the item was presented. 

^Objective code. The number of the objective this item represented was recorded. 

^ ^Previous encounters This numbered the times the same item was previously presented within the same lesson. 
^Previous correct. This indicated the number of consecutive correct responses to previous presentations of the same 

item. 

^Intervening items. The difference between the current trial number and the trial number of the immediately 
preceding presentation of the same item was indicated. (Note: for the first presentation of each item, this number was 
meaningless since it was the same as the trial number.) 

'iVob. Correct, This was the probabflity of a correct response to the current presentation. 

^Response, A "1" indicated a correct response and a "0** indicated an incorrect response to the current item. 

^Previously mastered. This indicated the number of items mastered before the current trial. 
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Simulated Drill and Practice Student Protocol Report 
Part 3: Item Summary 



Item^ 




# of ^ 












3 


5 


2 


100 


4 


5 


4 


80 


5 


4 


4 


55 


6 


4 


2 


100 


8 


2 


3 


66 


9 

• 


5 

• 


6 

• 


66 

• 


• 

44 


• 
• 

2 


• 
• 

5 


• 

60 



Item, The number of the item is given. 

^Obj\ code. The number of the objective to which the item belongs is identified. 

# of responses. The number of responses to the item up to and including the response at which mastery was 
attained. 

^% correct. The percent of the above responses which were correct. 

TABLE 4 

Simulated Drill-and-Practice Student Protocol Report 
Part 4: Instructional Objective Sunmary 



Obj/ 
Code 


2 

# Of 

Items 


# of ^ 
Responses 


4 

Presentations 
Per I tem 


1 


1 


2 


2.00 


2 


3 


15 


5.00 


3 


7 


21 


3.00 


4 


7 


26 


3.71 


5 


12 


64 


5.33 


Total ^ 


30 


128 


4.26 



^Objective code. The number of the objective summarized in that row. 
^# o/ items. The number of items assigned for the given objective. 

responses. The total number of item responses for the given objective. 
^Presentations per item. The number of responses divided by the number of items. 

^Totd, This gave the total number of items, the number of responses up to mastery, and the mean number of 
presentations per item for the lesson. 
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TABITT 



Simulated ')rill and Practice 
Report to the Student Before Lesson 



AFSN 


NAME COURSE 


BLOCK 


LESSON 


INSTRUCTOR 


DATE 


768901 


Jones , A. E. PME 


III 


01 


Clark, D. 


08/24/72 



Lesson Topic- 
Resonance In tuned LRC circuits 

*** Before Lesson*** 

***Ass1gnment*** 

The nurrber of Items you have been assigned 1s- 
30 

2 

The estimated number of minutes for you to complete this assignment is- 
60,C 

^Assignment This indicated the total number of the items prescribed for the student. 

2 • • • 

Time estimate. This was the estimated time (in minutes) needed for the student to reach mastery of all the items. 
The time for review was excluded. It was assumed that the average student needs six trials to pass each item and 20 seconds 
for each trial. 
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Simulated Dril l»and«Pract1ce 
Report to the Student After Lesson 



AFSN 
768901 


NAME COURSE 
Oones, A.E. PME 


BLOCK 
III 


LESSON 
01 


INSTRUCTOR 
Clark, D. 


DATE 
08/24/72 




*** After Lesson^ *** 










ff OT 

Items 


#of^ %" 
Trials Correct 


Time 
(Min.) 


1 






30 


128 57.4 


43.6 










6 

% Correct for Each Objective 








Obj.l 
45.4 


Obj.2 Obj.3 
48.6 53.3 


Obj.4 
68.4 


Obj.5 
56.6 







^ After lesson. This represented the overall result of the student's performance in the lesson. 

^# o/ items. This indicated the total number of items the student completed during the lesson. In general, the 
student should complete all the items in the lesson. 

^# o/ truds. This indicated the total number of trials attempted by the student. 

^% correct. This shows the ratio between the number of successful trials to the -^al number of trials. 

^Time. The total time spent by the student in the lesson is recorded in minuses. 

^% correct for each objecth^. For each objective the percent of the responses that were correct was shown to 
indicate the student's strengths and weaknesses. 
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TABLET 



Simulated Drill-and-Practice 
Report to the Instructor 



COURSE 
PME 



LESSON 
01 



LESSON TOPIC 
Resonance in Tuned LRC Circuits 



Dates ■ 
08/24/72-08/31/72 


Instructor 
D. Clark 


Enrollment 
15 




Obj.l 
4 
31 

• 
• 


Obj.Z 
63 
98 

• 
• 


Pretest Results 
Obj.3 Obj.4 
99 99 
71 55 

• • 

• • 


2 

{% correct) 

Obj.5 Total 
75 75 
84 73 

• • 

• • 


AFSN 
898165 
449883 

• 
• 


• 

Means 

37 


• 

56 


• 

66 


• 

56 


• • 

51 55 


• 


Standard Deviations 
18.0 27.1 


27.1 


29.0 


Oc,»C JLO.D 




Obj.l 

100 

100 

• 
• 


Obj.2 
100 
55 

• 
• 


Performance Summary 
Obj.3 Obj.4 
66 100 
100 100 

• • 

• • 


3 

(% correct) 

Obj.5 Total 
75 86 
83 78 

• • 

• • 


AFSN 
506940 
898185 

• 
• 


• 

Means 
62 


• 

73 


• 

77 


• 

69 


• • 

70 64 


• 


Standard Deviations 
24.3 19.5 


20.1 


19.7 


16.0 10.4 





^ Dates, Individual students may have taken the lesson on any day within the specified period. 

^Pretest results. The percent correct on the items for each objective, as well as the percent correct for the total 
pretest, was given for each student. Since there were varying numbers of items per objective on the pretest, the total score 
was a weighted average of the objective scores (calculated from the objective selection index, Table 1). The students were 
ranked from high to low on the basis of thei/ total pretest score. The means and standard deviations for each objective and 
the total pretest were given. 

^Performance summary. This section of Table 7 gives the percent correct of each student's responses up to and 
including mastery on each item for each objective and for the total drill. The means and standard deviations are also given. 
Again the students were ranked from hi^ to low on the basis of their percent of correct responses in the drill. 



ERIC 



23 



TABLE 8 



Simulated Drill-and-Practice 
Management Report 



COURSE 


BLOCK 


LESSON 


DATES 


REPORT NUMBER^ 


INSTRUCTOR 


PME 


III 


01 


08/24/72-08/31/72 


02 


Clark, D. 



Participating Students 

569547 203800 376074 



899106 396353 123656 536049 



Number of Students 
45 



Mean 
S.D. 



Mean 
S.D. 



Mean 
S.D. 



Obj. 
50 
27 



1 



Obj.l 
2 
1 



Obj.l 
72 
23 



Obj .2 
46 
29 



Pretest Results (% correct) 

Obj. 3 Obj .4 Obj. 5 Total 
49 50 46 48 
26 27 29 28 



Assignment (# of items) 
Obj. 2 Obj. 3 Obj. 4 Obj .5 
3 3 3 5 

1113 



Total 
16 
6 

6 

Performance Summary {% correct) 
Obj. 2 Obj. 3 Obj. 4 Obj. 5 Total 
70 65 69 65 68 
20 19 19 19 20 



^Report number. These reports were numbered sequentially for ease in filing. 
^Participating students. The AFSNs of the students included in the report were listed. 
^Number of students. This indicated the number of students included in this report. 

^Pretest results. The means and standard deviations of die pretest scores for each objective were recorded in terms 
of percent correct. The total pretest result was a weighted average calculated using the objective selection index from the 
student protocol report (Table 1). 

^Assignment. The means and standard deviations of the number of items assigned for each objective and for the 
total drill were recorded. 

^Performance summary. The means and standard deviations of the percent correct of the responses during the drill 
up to and including mastery were recorded for each objecthre and for the total drill. The total was a weighted average 
(same as the pretest results). 
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TABLE 9 



Simulated DrilNand-Practice 
Cumulative Report 



COURSE 


BLOCK 


LESSON 


DATES 


REPORT NUMBER 


INSTRUCTOR 


PME 


III 


01 


07/15/72-08/31/72 


02 


Clark, D. 



Nunter of Students 
90 









Pretest Results 


( % correct) 




Obj.l 


Obj.2 


Obj.3 Obj.4 


Obj.5 Total 


Mean 


49 


49 


52 52 


49 50 


S.D. 


26 


29 


28 28 


30 29 








Assignment (# of 


items) 




Obj.l 


Obj.2 


Obj.3 Obj.4 


Obj.5 Total 


Mean 


2 


3 


3 3 


5 16 


S.D. 


1 


1 


2 2 


3 6 






Performance Summary 


{% correct) 




Obj.l 


Obj.2 


Obj.3 Obj.4 


Obj.5 Total 


Mean 


70 


69 


67 69 


68 68 


S.D. 


23 


20 


19 19 


19 20 



Item summary. The fifth simulated drill-and-praciice report, the item summary report, was requested 
after a large number of students had completed the lesson. For each item the difficulty index and the 
number of students who were assigned the item were reported. This report was intended for use by the 
instructional development and evaluation component of AIS to facilitate the evaluation of the instruction 
and also to identify any problems wWch should be replaced. Table 10 depicts a simulated Item Summary 
Report. 



Future Validations and Model Refinement 

Controlled study groups, ahernative model development, and new estimation procedures might be 
pursued to validate the simulation procedures. The FSU team recommends that an alternative strategy be 
utilized that both maximizes review mastery and minimizes training time. Masteiy levels can indicate the 
effectiveness of the prescription process and the criterion for problem elimination. Time saving should 
reflect the quality of the prescription and block size process. The extension of latency criteria and the 
elaboration of the task index will help interrelate the behavioral findings with the task structural features 
found in technical training. The simulated drill-and-practice model was designed using, in part, assumed 
parameters. To operationalize the model, these parameters, optimum block size, task selection index, index 
of studer t characteristics, criterion for mastery, and size of lesson, need empirical investigation. This should 
assist in the refinement and generalizability of the model. 
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TSBLT 10 



Simulated Drill-and-Practlce 
Item Summary 



COURSE 


BLOCK 


1 C^VlN 




DATE 


PME 


III 


ni 

U 1 




nft/31/7? 


Number of Students 








90 










1 




2 


3 


4 


Item 


Obj 


. Code 


N' 


% Correct 


1 




4 


30 


61 


2 




2 


44 


74 


3 




1 


32 


67 


4 




3 


40 


74 


5 




4 

• 


35 

• 


70 


44 




• 
• 

5 


• 

38 


65 



*/(em. The number of the item presented to the student is shown, 

^Oby. code. The number shown represents the objective to which the item belonged. 

^N. The number of students presented the item is given. 

^% correct The number represents the percent correct of all responses. 



III. PACING MODEL 

Pacing models are defined as instructional systems which control the amount of problem-solving time 
permitted a student based upon his performance. The g^gl of the pacing model is to increase 
problem-solving speed without jeopardizing accuracy. The student's accuracy or mastery level takes 
precedence over his problem-solving speed. However, if the learner's efficiency can be improved without 
adversely affecting his accuracy, then one of the major goals of AIS can be achieved; Le., learner efficiency. 

The pacing model was simulated to demonstrate the feasibility of having an adaptive instructional 
system in the AIS which increases problem-solving speed while not debilitating accuracy. This section is 
introduced with a description of the model's general characteristics and data elements. The second section 
presents the major components of the simulated pacing model and four types of output reports. The 
section concludes with a narrative on the validation and recommendations for future refinements. 

The pacing model, specified here, was designed for use within a wide range of problems or study 
tasks. It should be especially useful for reading, study, and problem-solving tasks of known difficulty levels. 
The objective of task flexibility was accomplished by varying the time. For example, if performance was 
high, the time allowed for problem-solving was decreased, and if performance was low, the student was 
given more time to solve the next set of problems. The pacing model used no data other than that generated 
by the student within the task context; that is, the decision structure of the pacing algorithm uses only 
individual, in-task performance data as input. Future developments of the model will include preassessment 
of task data as factors for accurately predicting performance effective presentation rates. The model will 
maximally adapt accuracy and speed within a sequenced task structure. 
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Data Elements 



The data e-.ements used in the simulated pacing to assess student behavior were task characteristics 
(operational variables of the task) and performance data (within-task behavior). Preassessment data; e.g., 
AQE and AFQT scores, educational level, were not used directly within the simulated pacing model. All 
input data used in the decision structure of the simulation came from within-task performance measures. 
The decision structure used a set of rules based on an individual's in-task performance data to arrive at a 
particular decision by a systematic and efficient means. 

Task characteristics. Since the pacing algorithm was adaptive to the individual's performance, and it 
was that performance which was affected by task difficulty, it followed that the algorithm was adaptive to 
task difficulty features. The algorithm worked best with homogeneous problem sets, but task difficulty per 
se needed no rigid control. The pacing model may be used in study or problem-solving sequences which are 
presented via a computer-controlled typewriter or display device. No other medium is as functional as the 
computer interactive terminal because performance reports must be presented to the student while he is 
taking a paced lesson. 

I-jrformance data. Within the pacing model two types of performance data were utilized to determine 
the amount of the increase or decrease in the time allowed for the next set of problems: (a) the number of 
correct re^onses, and (b) latency data (student time to respond) were used to determine whether the 
student's problem-solving time was to be increased, decreased, or stabilized. 

Simulation Structure 

The pacing model was selected for simulation from among the seven adaptive instructional models 
(Hansen et aL, 1972) because it not only improved effectiveness of the learning process, but also increased 
the efficiency of the process. The simulation used the APL computer language for use on the CAI Center 
1500 IBM system. The program simulated a total of 90 students to demonstrate sufficient output for 
analysis of the pacing model. 

M^or Components of Facing Model Simulation 

The components of the pacing model simulation were: (a) a set of algorithmic decision rules, 
(b) student performance data in the form of correct responses and latencies, (c) informative feedback to 
the student while he was performing the task, and (d) summative reports to personnel at the instructor and 
supervisor level. 

These components functioned together in the manner shown in Figure 4. 

When the first set of problems was presented to the student, he was given as much time as he needed 
to solve them. If his performance on this problem set was equal to or greater than 60 percent correct, he 
entered the pacing aJgorithm. If his performance was below 60 percent correct, he did not enter the 
algorithm, but was given a new problem set and allowed to work through it at a self-paced rate. 
Immediately prior to that presentation, the student was given a message encouraging him to improve his 
accuracy. Using this procedure, the student did not enter the pacing algorithm until he had achieved 
sufficient mastery of the material. Entering the algorithm prior to obtaining a degree of mastery was 
assumed to be maladaptive to both his future performance and problem-solving speed. 

When the student did perform at a 60 percent accuracy level or better, the pacing algorithm 
controlled the problem-solving lime allowed for the next set of problems. Several decision rules became 
functional at this time. 

1. If performance was greater tlian or equal to 80 percent correct, then the problem-solving time 
allowed for the next set of problems was set at the mean of (a) his current time allowed, and (b) the 
current mean latency (Tj). Initially, tthis decreased the amount of tin'.e allowed, yet it did not overtax the 
student's capabilities. Rather, it served as a stabilizing mechanism, whereby the problem-solving time was 
gradually reduced. 

2. If performance was equal to 60 percent correct, then the problem-solving time allowed for the 
next set of problems remained unchanged (T, ). 
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Figure 4. --Pacing Model Flowchart 



3. If performance was less than 60 percent correct, the problem-solving time allowed for the next set 
of problems was set at the mean of (a) the current allowed time and, (b) the next lower allowed time 
during which the student correctly answered 80 percent or more of the problems (T3). 

4. If the student obtained a 60 percent level of accuracy on two consecutive problem sets, his 
allowed time for the next set of problems was set at the next lower allowed time during which the student 
correctly answered 80 percent of the problems, rather than T3. 

5. The paced sequence was terminated when the student's performance (accuracy and speed) met a 
specified criterion. This criterion was arbitrarily set at a performance greater than or equal to 80 percent 
correct, when the absolute change in maximum time allowed was less than the larger of (a) two seconds or, 
(b) the standard deviation of the latencies of the last problem set. That is, the paced sequence was 
terminated when performance was hi^ and when there was no evidence of any further improvement in 
problem-solving speed. 

28 

ERLC 



At the conclusion of each problem set the student received a record of his performance, and feedback 
which informed him of the decision made by the computer to either increase, decrease, or stabilize his 
problem-solving time. When a student completed a lesson, a student protocol report was immediately 
generated. Similarly, when 45 students taking a particular lesson had terminated, a management report and 
an instructor report were generated. These reports were valuable for providing immediate knowledge of 
results to persons at all levels in the instructional process. With detailed reports available, irregularities 
within the global instructional system, as well as within the pacing model itself could be identified and 
corrected. 

Output Reports 

Four reports were designed for the pacing model simulation to illustrate the variety of information 
which could be coUected. These reports provided feedback to the student so that he could participate in the 
learning process; and data to the instructor for keeping an updated record on each student. Such data are 
useful and necessaiy for the maintenance and revision of the proposed AIS. The four sample reports from 
the pacing model simulation were: 

1. The Student Protocol Repoit 

2. The Report to the Student 

3. The Report to the Instructor 

4. The Management Report 

Student protocol report. The student protocol report provided a detailed description of a student's 
performance on a particu/ar lesson. The report gave data on three major sections: (a) initialization 
information; (b) response lecords on each problem set ; and (c) summaries on number of problem sets and 
learning latencies. These sections will be presented by a table of simulated data on one student with 
annotations of the tabled information. Table 11 shows an example of a protocol report simulated for a 
given airman. 

Student report. A report to the student was provided immediately after completion of a problem 
set. It included the number of problems seen, the number correct, the number of items the student failed to 
answer in the time allowed, and his average response time. In addition, feedback was given to the student 
regarding his current performance level, and suggestions were given about how he should respond on the 
next problem set. Table 12 shows an example of a pacing model report simulated for a given airman. 

Instructor report. To maintain current updates on student performance the instructor's report 
provided a student profile per lesson. Along with the initialization data, the report presented the mean 
latencies for the first and last problem sets per student. The difference between these two latency figures 
indicated to the instructor the amount of change in the student's problem-solving speed. Table 13 contains 
an example. 

Management report. Two supervisory reports were designed for the simulated pacing model to give 
summary data on the total lesson performance of currently enrolled students, and a cumulative analysis on 
all current and previously enrolled students. These reports were issued for lessons within blocks. The 
Management Report was issued immediately after enrolled students completed a lesson. This report (Table 
14) allowed the supervisor to review student group progress on the various lessons and blocks. Combining 
this knowledge with the Cumulative Management Report (Table 15) the supervisory personnel had the data 
necessary for locating potential revision areas. 

Validation and Future Refinements 

Validation of the pacing model sliould assess its effect upon reading effectiveness and efficiency- 
Co nsequently , there should be more use of the model during the early blocks of a course with fewer and fewer 
requirements during the later blocks. This would be a direct indication of the efficacy of the model. 
Secondly, the decision rule by which adjustments are made in the pacing presentation should be varied in a 
systematic manner so as to find the one which improves reading rates the most. Lastly, investigation can be 
pursued into the frequency with which the model is applied to given students, studying the long-term 
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TABLE 11 



Simulated Pacing Model 
Student Protocol Report 



AFSN NAME COURSE BLOCK LESSON INSTRUCTOR DATE 

768901 Jones , A. E. PME III 01 CI ark, D. 08/24/72 



Prob/ 


Tlme^ 


Number' 


Number"* 


Me art 


Latency* 


Change' 


Set 


A1 1 owed 


Timeouts 


Correct 


Latency 


S.D. 




1 


100.00 


0 


4 


25.31 


5.89 


-37.34 


2 


62.65 


0 


3 


21.51 


4.66 


0.00 


3 

• 


62.65 

• 


0 

• 


3 

• 


24.97 

• 


4.11 

• 


0.00 

• 


• 

32 


• 

6.47 


• 
• 

1 


• 
• 

4 


• 

4.'61 


• 

1.'40 


• 

-0.'93 



, Sumtiary n 

# of Mean Set Latency Change 
Sets First Last 

32 25.31 4.61 -20.69 



^ Problem set. The listing contains the problem sets within the lesson taken by the student. 

^Itme allowed. The maximum time limit expressed in seconds, within which the student must respond, is given for 
each problem set. 

^Number of timeouts. The number represents the rimes the student failed to respond within the time allowed. 

^Number correct. The number of questions correctly ansu'cred by ihe student is given for each problem set. 

^Mean latency. The .veragc response time (seconds) to the questions within a problem set is shown. 

^Laitmcy S.D.. The standard deviations of the student's response times are gh^en per problem set. 

"^Change. The number shown represents the amount of the increiM or decrease in the time allowed from one 
problem set to the next. 

^Summary. These culujnns summarize the protocol report. 

^ ^ of sets. The number given is the total problem sets encountered. 

^^Mean set latency (first-last). The mean response latency is given for the first and last problem sets. 

' ^Change. The figure represents amount of the decrease in response latency from the first to the last problem set. 



effects on their reading and problem-solving rates. The goal is to determine the algorithm which promotes 
increase in problem-solving speed and has the most lasting effect. In the future, decision algorithms should 
be considered that incorporate variables such as a student's confidence in his performance and reading rate. 
The incorporation of preassessmcnt variables also seems appropriate. 
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TABLE 12 



Simulated Pacing Model 
Student Report 



AFSN 


NAME 


COURSE 


BLOCK 


LESSON 


DATE 


768901 


Jones ,A.E . 


PME 


III 


01 


08/24/72 



This is a sunmary of your recent performance 

12 3 4 

Number Number Number Average 

of Correct of Response 

Items Timeouts Time (Sec) 

5 3 0 25 



You only answered 3 problems correctly that time. 
I will give you the same amount of time for the 
next set of problems. Try to get them all right 
this time.^ 



^ Number of items. The number of items in the immediately preceding problem set is given here. 

^Number correct* The number of problems correctly answered is shown. 

^Number of timeouts. The number of times items were not answered in allotted time. 

^Average response time (sec). The mean response latency of the immediately preceding items is expressed in 
seconds. 

^Feedback is given about performance and the state of the pacing procedure. 
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TABLE "13 



Simulated Pacing Model 
Instructor Report 



COURSE BLOCK LESSON DATES INSTRUCTOR 

PME I 01 08/11/72-09/03/72 Clark, D. 



1 

Number of Students 
15 



AFSN^ 


# of ^ 


4 

Mean Set Latency 


Change ^ 




Sets 


First 


Last 




692304 


13 


31.79 


11.90 


-19.89 


607494 


14 


23.14 


11.44 


-11.70 


912525 


17 

• 


19.36 

• 


10.12 


- 9.24 


• 

947851 


• 

45 


• 

28.33 


• 

2.11 


• 

-26 '22 


Mean^ 


29 


26.68 


5.71 


-20.96 


s.dJ 


12 


4.68 


3.36 


5.97 



Number of students The total number of students included in the report is given first. 

^Al'SN, The IdentiQcation number of each participating student is listed. 

of sets. The total number of problem sets taken by each student appears. 

^Mcan set latency (first-last). The mean response latency if gh^en for the first and last problem sets for each student. 

^Oiange, The figure represents the amount of the decrease in mean response latency from the first to the last 

problem set for the individual. 

6 * 
Mean, TTie columns show the averages of the number of sets, mr^n set latency (first and last problem sets), and 

change scores. 

Columns show the standard deviations of the number of sets, mean set latency (first and last problem sets), 
and change scores. 
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TABLE 14 



Simulated Pacing Model 
Management Report 



COURSE 


BLOCK 


LESSON 


DATES 


REPORT 


PME 


II 


01 


08/11/72-08/12/72 


no 


Number of Students^- 








45 












# of^ 


3 4 

Mean Set Latency Change 






Sets 


First 


Last 




Mean 


25.51 


25.93 


6.57 -19.36 




S.D. 


10.90 


4.51 


3.49 5.65 





Number of students. The cumulative total number of students who completed the lesson is shown. 



# of sets. The column presents the mean and standard deviation of the number of problem sets completed by the 
students enrolled. 

^Mean set latency (first-last). Included is the mean and standard deviation of the first and last mean problem set 
latencies. 

^Change, The mean and standard deviation of the latency change from the first to the last problem sets. 

TABLE 15 

Simulated Pacing Model Cumulative 
Management Report 



COURSE BLOCK LESSON DATES REPORT 

PME III 01 08/11/72-09/12/72 02 

Number of Students^ 
90 





# of ^ 


Mean Set Latency' 


Change"* 




Sets 


First 


Last 


Mean 


26.12 


25.46 


6.37 


-19.09 


S.D. 


11.31 


4.61 


3.43 


5.64 



^ Number of students. The cumulative total number of students who completed the lesson is shown. 
studentstS^M* presents the mean and sundard deviation of the number of problem sets completed by the 

latencici^''''" ^^''"''^ ^M^-W;. Included is the mean and standard deviation of the first and last mean problem set 
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^Change, The mean and standard deviation of the latency change from the first to the last probli 



em sets. 



IV. MONITORING MODEL FOR ADAPTIVE INSTRUCTION 



Adaptive instructional models have different purposes and techniques. The drill-and-practice model 
described previously, for example, uses preassessment measures and rules to guide the student to mastery 
after instruction has occurred. The pacing model simulation uses only wlthin-instruction measures to 
achieve faster performance times without degrading the student's mastery. As the AIS is developed and 
becomes operational, other models will be defined with similar context-specific rules and goals. 

After implementation of AIS instructional materials with such models, it will still be necessary to have 
techniques for selecting from among these the most probable alternative for student success. It will be 
necessary to decide when and for whom these models for instruction will be used. The monitoring model 
provides this capability by relating measures of each student's unique learning attributes to the instructional 
options and the predicted availability of resources. 

The monitoring model has as its primaiy purpose the selection of instructional treatments via 
prediction techniques that match characteristics of individual students with training characteristics. Each 
treatment is composed of media types, instructional topics for content, instructional strategies, and other 
aspects of the instructional environment which may be relevant in the training process. The model is 
algorithmic because it uses a set of rules to arrive at the individualized instructional treatment. The 
algorithmic rules applied in this model were based on multiple linear regression analysis and sequential 
decision making. Regression techniques ^tttempt to regress scores toward a central point along a line fitted 
to the scores. Regressing across multiple predictor variables such as prior training, ability scores, personality 
variables, aptitude tests, and historical variables aids in finding the optimal location on the line. That is, 
given student scores and task characteristics, the performances and times of several alternative instructional 
treatments could be predicted for a given student at a given task. It is necessary to have a regression 
equation, consisting of predictor variables into which particular scores for students are input and from 
which predictive outcome values can be generated, for each alternative training treatment. Comparing the 
predictive values produced for each alternative treatment makes it possible to select the best treatment 
predicted for an individual. In the computer-based environment of AIS, the student could be assigned his 
treatment directly '/ia an AIS terminal. 

Since one of the major tasks for AIM in designing adaptive models was to determine appropriate 
variables with predictive value and to continuously monitor their effectiveness, the simulation approach 
demonstrated a technique wfthin the model which categorized the types of variables according to their 
expected interrelated predictive values and their methodological updating requirements. The predictor 
values of personality, aptitude, and performance/state variables were separated into three different 
regression equations for each instructional treatment. In this manner, the utility of each type of predictor 
measure was updated and evaluated more easily. In addition, within the computer-based environment of 
AIS, it will be possible to change the predictor variables within a given type (e,g,, personality, aptitude, or 
performance/state) and to update the values of the equations more data are acquired on the students 
trained by AIS. 

Data Elements 

In this section the variables required for input and output are identified and defined. Input variables 
are categorized as preassessment data (measures of the student or task taken prior to lesson participation 
for use in the prediction) and macro-performance data (measures of student performance and time taken 
from lesson participation) which may be used to modify the learner's behavior through incentives or 
updating of predictors. Output variables are discussed within the context of the inputs for both 
preassessment and macro-performance data, since the outputs are the purpose of the inputs and are not 
independent. 
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Pteassessment data. The monitoring model simulation used regression equations with predictor 
categories consisting of aptitude, personality, and performance/state variables to select instructional 
treatments. The goal was to predict the best perfomiance for a given student on a given task with 
alternative instructional treatments. It was necessary to have a regression equation for each type of 
treatment into which particular values for students could be input, and from which a predicted outcome 
could be generated. Comparison of the statistical values for each treatment permitted selection of the best 
predicted treatment for an individual. From the initial regression weights and the corresponding student 
data, the estimated time and performance scores for each student were computed. In addition, these 
regression coefficients were assumed to be updated periodically as more data were collected on the 
students. For purposes of this simulation, the beta weights and student data were generated utilizing 
correlation matrices for all of the predictor variables. The predictor variables selected for feasibUity and 
simulation were grouped into three categories. 

Group 1, aptitude variables, were basic capabilities that the student brought to the course. Hence, 
aptitude variables exemplified measures which would likely have valuable predictive power with student 
adaptive instructional treatments. Four of the variables chosen were from tests taken by Air Force enlistees 
(ASVAB word test and arithmetic reasoning test, AQE electronics aptitude test and AFQT). A fifth 
variable, task index, was a measure specifically related to the behavior involved in the performance of AIS 
lessons. The task index ranged from a low score, reflecting abstract, symbolic though, to hi^h scores 
reflecting psychomotor performance tasks. The task index reflected the close relationsliip of task 
requirements with media features of the treatment. 

Group 2, personality variables, consisted of five personality measures assumed to have validity for Air 
Force students enrolled in the PME course. The first, need achievement, was an estimation of the student's 
motivation or drive. The second, trait anxiety, was a measure of a consistent level of anxiety over a period 
of time as opposed to momentary anxiety caused by specific situations. The third variable, epistemic 
curiosity, was a measure of the student's search behavior for further information. The fourth personality 
variable, media adaptability index, indicated the capability of the student to acquire information by a type 
of instructional media presentation. The final variable was that of learning style, which indicated the 
dominant strategies a student used in a learning task. 

Group 3, performance/state variables, were closely related to the macro-measures to be discussed in 
the next paragraph (score and time on previous lesson, pretest score, state anxiety, and incentive average to 
date). Performance/state variables were measured just prior to or between lessons, and were updated 
between lessons. This was unlike the variables of the first two groups which were fixed prior to instruction 
to a complete block or course. 

Macro-performance data. The macro-performance data consisted of measures taken between lessons 
which were primarily used to arrive at proper incentive and lesson management. In lesson B the incentive 
units were generated according to an algoiithm which provided more incentive units to the student who 
spent a longer time on the previous lesson than had been targeted. This algorithm was an attempt to 
motivate the student by telling him he would gain the larger number of incentive units if he went faster 
during the next lesson. Thus, his time on the previous lesson was the input to this incentive management 
technique. The output was the number of incentive units to be provided. The algorithm follows: 

1. If a student took between zero and five minutes over the predicted time for lesson A, he would 
acquire a multiple of two times the number of standard units for lesson B. 

2. If a student took from five to ten minutes over the predicted time for lesson A, he would achieve a 
multiple of four times the standard incentive units for lesson B. 

3. If a student spent more than ten minutes he would achieve a multiple of six incentive units over 
the standard number for lesson B. 

Simulation structure. The monitoring model, using regression techniques, has been selected as an 
overall monitor model for the adaptive models of AIS. A function of the model is to maintain control on 
available resources. Another functional mode is the collection of lesson performance and time data for 
student prescription. This could also be used in evaluation, both formative and summative. In effect, the 
model's goal is training time reduction by analysis, prediction, and selection of mastery-oriented 
instructional tasks. The role of the monitoring model is to insure that cost effectiveness and training time 
reduction are realized. 
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Major Components of the Monitoring Model Simulation 

The general components of the multistage monitoring mode! are discussed in three stages: 
(a) analysis, (b) prediction and monitoring, and (c) evaluation and update. The analysis stage was a time 
period in which the initial measures for predictor variables were chosen on the basis of the empirical 
literature. Measures on trainees at Lowry Air Force Base produced data for analysis in regression equations 
to determine the most appropriate combination of predictors for use within the computer-based model 
during real 4ime train ing (Figure 5), 
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Figure 5. —Functional flow of the Monitoring Model 
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Th^^ second stage, prediction and monitoring, began with these equations and predictor variables. As 
each student proceeded through lessons, his assessment measures and instructional history measures were 
input to these equations along with trie existing beta coefficients of the previously determined equations. 
Criterion values by which to select treatments were generated from the equations by predictor-type groups 
for both perfonnance and time and for each alternative. The groups' criterion values in the model would be 
summed for both performance values and time values. The next step was to select the treatment on the 
basis of these rankings. If no treatment had a predicted mastery, the one with the best predicted 
performance was selected. If the student failed this selected treatment, the next best treatment was chosen. 
If the student failed all selected treatments, the instructor was notified. For evaluation and updating of the 
equations and measures, the data for student performance and time were recorded and blocked according 
to the treatment chosen and student characteristics. In this way, both the decision rule for selection of a 
treatment and the treatment's effectiveness were evaluated. If the treatment was not used or was used 
ineffectively, it was modified or deleted from the instructional environment. 

Instructional context. The instructional context for the monitoring simulation was Block IV of the 
Precision Measuring Equipment course, titled, "Vacuum Tubes and Solid State Principles and Power 
Supplies." Unit 1, Electron Tube Theory, targeted for twelve hours of conventional instruction, was 
specifically chosen for the simulation. This unit was modified to contain five lessons. The best treatment 
within each lesson for any given student was predicted. It was necessary to defirie possible treatments 
within each lesson according to what seemed reasonable in terms of content and behavioral objectives as 
stated in the Air Force Plan of Instruction. The time and performance criterion values were generated for 
each of the following: 

Lesson A. Electron Emission (1 hour) 

1. Text and workbook 

2. PI text and workbook 

3. Film loop and workbook 

4. Television 

5. CAI tutorial 

Lesson B. Tube Filaments (.2 hours) 

1 . Text and workbook 

2. PI and text and workbook 

3. Film loop and workbook 

4. Television 

5. CAI tutorial 

Lesson C. Diode Tubes (6 hours) 

1. PI Text 

2. Programmed slide tape presentation and workbook 

3. PI text and computer-based tutorial with drill and practice 

4. Text film, and workbook 

5. Film, workbook, and computer-based tutorial and drill and practice 
Lesson D. Tube Tester Usage (1 hour) 

1 . Concept film, study guide (redundancy level 1), and workbench 

2. Concept film, study guide (redundancy level 2), and workbench 

3. Concept film, study guide (redundancy level 3), and workbench 

4. Instructor demonstration, study guide (redundancy level 1), and workbench 

5. Instructor demonstration, study guide (redundancy level 2), and workbench 
Q 6. Instructor demonstration, study guide (redundancy level 3), and workbench 
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Lesson E. Gas Diodes (2 hours) , , ^ 

1. Slide tape, workbook, and workbench 

2. Student guide, workbook, and workbench 

3. Videotaped lecture, workbook, and woricbench 

4. CAl tutorial, workbook, and workbench 

5. PI text, workbook workbench 

Each of the five simulated Irascnshad five or more instructional alternatives (more alternatives could 
be handled within the model). A total of 26 instructional treatments were considered in the simulation, and 
regression equations were generated for each alternative treatment. For all lessons a best treatment was 
selected maximizing performance witli a minimum time. 

Incentives. The simulation illustrated the use of incentive management techniques throughout all 
lessons. The student was provided with incentive units (tokens) which could be used to "buy" desired 
outcomes such 'do time off base, higher pay and promotion. The simulation used an arbitrary time generated 
by the prediction of time as tfie expected average for completion by a student. For each one minute he 
gained, an incentive unit was given to the student. At the end of each lesson, the student was told the 
number of incentive units he gained or lost. These arbitrary formulas will be replaced in the operational AIS 
by empirical determination. 

Output Reports 

Three reports were designed for the monitoring model simulation. Three sample reports from the 
simulation will be presented and discussed. They are: 

1 . The Report to the Student 

2. The Report to the Instructor 

3. The Management Report 

The report to the student. Each simulated student received a report before he began each lesson, and 
after completion of the lesson. Table 16 shows that complete identification of the student, couise, and date 
were printed out prior to the assignment of instruction. Following completion of the lesson, all 
identification information was once again printed out for the student, plus the "after lesson" report as 
shown in Table 1 7. 

If the student in Tabie 17 had not reached mastery, his message would have read: "You have received 
a 78% score for lesson A, Electron Emission. This is failing. Ypu seem to be having trouble with this 
material. It is suggested that the following people in your class ma> \ e able to help. 

Creamer, U.K. 
Yaegpr, P.H. 
P&ul,T.H. 
Trotman, R.S." 

Report to the instructor For all students currently finishing lessons, the instructor received a report 
on the lesson containing the information simulated in Tab^e 18. The report contained a summary, a section 
on students failing to reach mastery, and individual records. 

The management report. The management report demonstrated the overall monitoring capability of 
the model. Management reports were generated per lesson, and were divided by instructor and student 
population per instructor. The assumption was that this report would go to a manager who had a number of 
instructors under his supervision. Table 19 is a simulated management repoit section for one instructor 
only ; the actual report would be composed of sections for a number of instructors. 

The management report was the final, and most comprehensive, report simulated for the monitoring 
model. A commentary on validation of the model concludes this section. 
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TABLE 16 



Simulated Monitoring Report to the 
Student (Before Lesson) 



AFSN NAME COURSE BLOCK LESSON INSTRUCTOR DATE 

224726 Frost, CP PME IV A Hiyt, RE 10/22/72 

Your assignment is 

First you should sign on to a computer terminal and read lesson 5. This 
wil I help you improve your reading speed. Next read Chapter 6 on electron 
emission in the electronics textbook TB 371. Finally, do exercises 1-7 in 
workbook WB 3ABR32420-17.^ 2 
Estimated time for this assignment is 1 hour. 

You may receive 1 incentive unit for each minute saved in addition to 5 
incentive units for receiving a score of 80% or better.^ 



The name of the assignment and its resource location. 

The estimated training time. 

The number of incentive units obtainable. 

TABLE 17 

Simulated Monitoring Report to 
the Student (After Lesson) 



AFSN NAME COURSE BLOCK LESSON INSTRUCTOR DATE 

224726 Frost, CP PME IV A Huyt, RE 10/22/72 

You have achieved a score of 91% mastery for lesson A) Electron Emission. 
This is passing and merits 5 incentive units. Your assignment was com- 
pleted in 31 minutes, a gain of 29 minutes, which merits a total of 34 
incentive units. Incentive units earned may be applied to any of the 
incentive items on the list passed out to you for this course. 
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TABLE 18 



Simulated Monitoring Report 
to the Instructor 



INSTRUCTOR COURSE BLOCK LESSON DATE 

Huyt, R.E. PME IV A 10/27/72 

Sumnary ^ 

Students passing = 14 Average Mastery Score = 84 Range = 80 to 92 



Students Who Failed to Reach Mastery on First Try 



Name 




AFSN 


Average Score 


Average Time 




Danner, 


D.E. 


323237 


78 


38' 




Studtnt Individual 


3 

Records 








AFSN 


Treatment Rank 


Mastery Score 


Time Incentives 




Assigned 


Order 


Predl cted/Obtal ned 


Predl cted/Obt. 


Gained 


873586 






91/88 


54/38 


27 


780538 






84/81 


48/50 


15 


220550 






86/82 


50/70 


5 


935596 






88/92 


51/48 


17 


423998 






85/81 


49/64 


b 


323237 


4 




81/78 


64/38 


0 


323238 

• 


1 

• 


2 

• 


84/80 


47/48 


17 

• 


• 

417767 


• 

4 


• 

i 


• 

81/84 


• 

45/31 


• 

34 



^Summary. This section of the instructor report presented the number of students passing, the average score, the 
score range, and the average time (minutes). 

^Students Who Failed to Reach Mastery ott First Try. These students, listed by name and AFSN, with average score 
and time arc shown in the student individual records as repeating the lesson via a different treatment (sec AFSN 323237). 

Student Individual Records. Six columns of information identified the student by AFSN, gave the number of the 
simulated treatment prescribed followed by the rank order of that particular prescription, mastery score predicted and 
obtamed, time predicted and obtained, and incentives gained. AU treatments prescribed, whether available or not, were 
listed in the simulation. 



40 



TABLE 19 ..... 
Simulated Monitoring Model Management Report 



COURSE 
PME 



BLOCK 
IV 



LESSON 
08 



DATE 



Instructor = Huyt, R.E. 

Number of students = 15 
Students Passing = 14 
Students Failing = 1 

Treatment Selection^ 



Shift = 1 



Average Mastery Score = 88 
Average Mastery Score =71 



(Range 81 to 95) 
(Range 71 to 71) 



Treatment Percent of Total Prescribed 
1 0% 
?. 25% 

3 0% 

4 69% 

5 6% 



Actual Number Prescribed 
0 
4 
0 
11 
1 



Treatment Unavailability 



Treatment 



Percent Unavailable 
of Selections 



Actual No. Unavail- 
able of Selections 



No. "Down" No "In Use" 
of Selections of Selections 



1 
2 
3 
4 
5 



0% 
43% 
0% 
0% 
0% 



Treatment Mastery Summary 

Treatment Percent Mastery Reached 
on Treatment 



0 
3 
0 
0 
0 



0 
3 
0 
0 
0 



Incentive Gains 
on Treatment 



0 
0 
0 
0 
0 



1 0% 

2 100% 

3 0% 

4 91% 

5 100% 

Total Incentives Given ^ 



0 
144 

0 

395 
49 



This unit = 588 

Average Incentives Given = 39 
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Treatment Selection, For each of the treatments available for prescription, the percent of prescription was given 
(ie., 25 percent of all students received prescriptions for treatment 2). The actual number of students receiving the 
prescription was also printed out. 

^Treatment Unavailability » For each treatment, the percent and actual number of selections unavailable were noted, 
and the reasons for unavaflability ("down" or "in use") completed the information on treatment availability. 

^Treatment Mastery Summary. This information gave the percent of the time that each treatment produced mastery 
(ie., treatment 4 produced mastery 91 percent of the times it was prescribed). The summary also presented incentive gains 
for each treatment. 

^ Total Incentives Given. Incentive information was in two parts: a total of incentives given to al! students on this 
unit, and an average number of incentives given per student for this unit. 
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Validation and Future Refinements 

In terms of future extensions, the most important validation approach would be to differentially 
weigh the sets of equations according to the cognitive, personality, and on-task sets of variables. These new 
functional relationships would provide a better fit between the predicted and observed values for given 
students. Other variations, such as the inclusion of new variables, new instructional treatments, and new 
resource arrangements, should improve the efficacy of the model in terms of its monitoring and managing 
capability. These are pathways to be pursued. 

Students* predicted and observed performance and time values should given a direct daily estimate as 
to the efficacy of the model. The establishment of appropriate beta weight estimates will require a large set 
(sample size in excess of 300) of students who will pass through the material witliout adaptation. As 
different conditions, such as material updates or modification are introduced, the findings wOl have to be 
updated to be relfected in each of the beta weights. As these periodic update values converge, one can be 
more confident that the model is working appropriately. 



V. CONCLUSION 

The previous three sections of diis report provided detailed descriptions of the computer simulation 
of three adaptive instructional models - drill-and-practice, pacing, and monitoring. Several steps were 
followed in the discussion of each model. First, there was a brief introduction to the purpose of the model. 
Next, data elements were specified. For each type of data needed, specific examples (AQE, AFQT, ASVAB 
scores) were used by the simulation to illustrate their roles within the model. Other measures should also be 
considered to fulfill these roles. (Similarly, the reports described in the previous sections suggested the 
types of information which can be generated by the models, but may be modified according to needs.) The 
third step was to flowchart the major components of each model. This provided an organizer for both the 
development and discussion of the models. The quantitative processes of the models as well as the 
mathematical functions used to generate student data for the simulations were discussed in this section. 
Finally, sample output reports were provided with annotated explanations of their content. Each model 
provided reports for the student, the instructor, and the management personnel. These reports suggested 
the types of information which can be generated by the AIM models. 

A major purpose of developing the simulations was to concretize the models developed in Phase I of 
the AIM contract. The contrast of details between the Phase I report which discussed the models in general, 
and the more detailed simulations described in this report, are highly visible. First, precise data needs were 
clarified. The operational processes were detailed, resulting in the computer programs documented in 
another report (McMurchie, Bennison, & Tam, 1973). Similarly, the possible output from each of the 
models was identified in detail. Furthermore, during the process of developing the simulations, flows and 
omissions in the original model designs were noted and corrected accordingly. 

Another outcome of the simulations was the identification oi t Iditional research which needs to be 
done in order to fully implement the models. First, the most appropriate measures to fulfill the input data 
needs of the models must be determined. Research is needed to investigate model parameters such as the 
optimum block size for the drill-and-practice models. In addition, the monitoring model requires an 
extensive data collection phase prior to its actual implementation. A large number of students should be 
randomly assigned to the available treatments to empirically derive the initial beta weights for the 
regression equations. Each of the preceding tliree sections concluded with suggestions for future validation 
and model refinement. 

The development of Adaptive Instructional Models (AIM) is a major step toward the realization of 
the Air Force's planned Advanced Instructional System (AIS). The AIMs hold the most promise for 
achieving the AIS goals of reducing training time without sacrificing performance levels. The simulation of 
the models proved to be a profitable step toward their implementation. It is, therefore, recommended that 
the same procedure be repeated with the other models described in the Phase I report, specifically the 
concept acquisition, rule learning, and problem-solving models. 
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